40. Tokenization
Language models operate on tokens rather than raw text.
A tokenizer converts:
text
↓
tokens
↓
token IDs
Possible tokenization approaches include:
- character tokens
- word tokens
- subword tokens
- byte-pair encoding
The vocabulary maps tokens to integer IDs.